v/Hir 



1/25 



FIG. 1 



13 



MPEG-4 
DECODER 



CLIENT 
12 



MPEG-4 
ENCODER 



-10 



14 



LAN 





STREAMING 
SERVER 



T 
11 



2/25 




3/25 



FIG. 3 



RTSP Server 



21 



RTSP Client f 



25 



TCP listen ()} - 



connectQ 



( accept () & create handler) 
connect succeed() j 

t . 

setup() 



c 



create RTP/RTCP \ 
Transport Engine Client J 



delete RTP/RTCP 
Transport Engine Client 



quit 



RTSP Handler 



_OK(). 
playQ 



(syntax checking and initialization) 





OK() 




f create RTP/RTCP 
^Transport Engine Server^ 




keep alive() 






OK() 






teardown() 


— -j 



OK() 



(stop data streaming, delete 
RTP/ " ' " ' 



VRTCP Transport Engine Server 



(quit the handler) 



4/25 




5/25 




6/25 




7/25 



FIG. 7 



Extended RTP 

32 bits 



V=2 



CC 



M PT 



Seq Num 



TimeStamp 



SSRC 



define by profile 



length 



TxGOVSeqNum 



tr 



cr 



SENTT 



dwGOVDataBuffSize 



dwGOVDataSize 



dwPreviousGOVBitrate 



dwCurrentGOVBitrate 



dwCurrentGOVPTS 



dwCurrentGOVDuration 



dwCurrentGOVSeqNum 



dwNumOfFrame 



dwCurrentGOVTransmissionSeqNum 



GOV data 



8/25 

FIG. 8 



Fields in The Extended RTP Packet 



Field 


Size 
\ UI1 / 


Function 


\/_0 


o 


MS IN nrO 1 ooy 


D 

r 


A 

1 


Ac in DPP1QQQ 

AS in nrL/looy 


X 


1 


As in RFC1889 


cc 


4 


AS in HrUlooy 


M 


A 

1 


AS in HrUlooy 


PT 


7 


As in RFC1889 


Seq Num 


8 


As in RFC1889 ; 


Time Stamp 


32 


As in RFC1889 


SSRC 


32 


As in RFC1889 


Profile 


16 


Extended Profile 


Length 


16 


Extended header length 


TxGOVSeqNum 


32 


GOV Sequence Number used 
in Transmission 


Tr 


16 


Total RTP packet for this GOV 


Cr 


16 


Current RTP number for this 
GOV 


SENTT 


32 


Server Estimated Next 
Transmission Time 


dwGOVDataBuffSize 


32 


The byte's number of the GOV 
data buffer 


dwGOVDataSize 


32 


The actual byte's number of 
GOV data 


dwPreviousGOVBitrate 


32 


The bitrate of last GOV 


dwCurrentGOVBitrate 


32 


The bitrate of current GOV 


dwCurrentGOVPTS 


32 


The starting PTS of current 
GOV " 


dwCurrentGOVDu ration 


32 


The duration of current GOV 


dwCurrentGOVSeqNum 


32 


The unique sequence number 
of current GOV 


DwNumOfFrame 


32 


The frame number of current 
GOV 


dwCurrentGOVTransmissionSeqNum 


32 


The transmission sequence 
number of current GOV 



9/25 



FIG. 9 



User Application RTCP 



32 bits 



V=2 



subtype 



PT 



Length 



SSRC/CSRC 



TxGOVSeqNum 



RTPacketNum 



Spare/Unused 



FIG. 10 



Fields in The User Application RTCP Packet 



Field 


Size 
(bit) 


Function 


V=2 


2 


As in RFC1889 


P 


1 


As in RFC1889 


X 


1 


As in RFC1889 


Subtype 


5 


As in RFC1889 


PT 


8 


As in RFC1889 


Length 


16 


As in RFC1889 


TxGOVSeqNum 


32 


Requested GOV Sequence Number to be 
retransmitted 


RTPPacketNum 


32 


Requested RTP packet number to be 
retransmitted, if equal to -1, retransmit the 
whole GOV 


Spare/Unused 


32 


Reserved for future used 



10/25 



FIG. 11 



Initialize internal parameters 



1^ 



S10 



Create main components, 
Push Timer object, CRTP 
object, CRTCP object, etc. 



On Timer trigger 



S13 



S11 



Set Timer trigger time to 0 



S12 



S23 



Set Timer to SENTT 



S15 




Read new 
GOV from 
buffer into 
tmp buffer 



S20 



S21 



S22 



Update 
present GOV 
information, 
bitrate, size 
...etc 



Calculate 
SENTT 



Sent 
RTP to 
Client 



S17 



S24 



S25 



Copy GOV data 
from tmp buffer 
to RTP 





Sent RTP 




Update 
internal 
parameters 




to Client 





S19 



S26 



Copy GOV data 
from tmp buffer 
to RTP 





Sent RTP 




Update 
internal 
parameters 




to Client 





S27 



11/25 



FIG. 12 



One Complete GOV 





New RG 


Middle RG 


Middle RG 


Middle RG 


Middle RG 


Last RG 



Cr = 0, 
Tr = 6 



Cr= 1, 
Tr = 6 



Cr = 2, 
Tr = 6 



Cr = 3, 
Tr = 6 



Cr = 4, 
Tr = 6 



Cr = 5, 
Tr = 6 



12/25 



2>cd 



O 

CD 
~o 
cd 

OS 

\ 
CD 
O 

> 

5 



a> 



TD CD 



<D 



CD 

CO 

a> 



CO 

CD 



O 
CD 

"O 

CD 

O 
CD 
Q_ 
X 



CD 

o 



£CD 



_CD 

^CD 
CD 



-52 

^CD 



5rn 



CD 
DC 



CD 
CO 



CD 

o 



o 

CD 

CO 

\ 

CO 
=3 

o 
> 

CD 



CD 

-o O 



CD 

IgCD 



CD 

-o CD 



CD 

cr 

CT3 



13/25 



Rx RTP packet 



S34 



FIG. 14 



Extract GOV infor 
of RTP packet 



S30 



Insert packet into buffer 




S37 



case 1 



S39 



case 2 



S41 



case 3 



case 4 



S44 



case 5 



S46 



case 6 



S48 



case 7 



S50 



case 8 



S52 



S54 



case 9 | - 



14/25 



CO 
CO 
CO 



cd DC 




15/25 



FIG. 16 



FIG. 17 



InsertGOV( ) 



Lock the Link Buffer. 
Forbid simultaneous access 



S100 



Remember the present 
inserting position and its index 



± 



S101 



1 




S111 



Prepare for calculating the 
Remaining GOV Playback 
Time and the Incomplete 
GOV N umber 

♦ ~ 



S120 



Get the node pointer for 
inserting 



S102 



Check whether the node's data 
buffer size is large enough to hold 
the GOV that will be inserted 



S104 




S103 



Backup the present values 
i 



S121 



Get the distance between 
the inserting pointer and 
the reading pointer 



S124 



Enough 
? 



NO 



S105 



YES 



1 



S106 




S122 



Re-allocate memory 
to the node 



Insert the GOV 



I 



S125 



Calculate the Remaining 
GOV Playback Time by 
skipping the Incomplete 
GOV and the Blank GOV 



Update the status of this GOV 
node (complete or not, etc) 



IT 



S107 



Update the array of indicating 
the receiving status of each 
RTP packet of the GOV 



S108 



S126 

Is the^ 
retransmission 
'allowed for the skipped 
Incomplete GOV 
or Blank GOV 
? 



NO 



YES 



S127 



The 

Inserting pointer 
overtakes the reading 
pointer 

S110 ^ "Tyes 



S109 



S111 



Overwriting happens, have to 
drop data and resynchronize 
the reading pointer with the 
inserting pointer 



Send retransmission request 
to the RTP/RTCP 
Transport Engine 



C return 1 



S128 



Calculate the Remaining 
GOV Playback Time and the 
Incomplate GOV Number 

— \ — 

( END ) 



16/25 



FIG. 18 



ReadGOV( ) 



S131 




S132 



( Return -3 J 



Lock the Link Buffer. 
Forbid simultaneous 
access 



S133 



S134 



S135 



Update the reading 
pointer and its index 



Get the node for 
reading 



Check whether this is 
a Complete GOV 




Copy the GOV out to the 
transmitted GOV node 



-S137 



Calculate the Remaining 
GOV Playback Time and 
the Incomplete GOV Number 



\S 



S138 



C Return the transmitted GOV node^ 



<D 



S139 



FIG. 19 




S138 



Prepare for calculating 
the Remaining GOV 
Playback Time and the 
Incomplete GOP 
Number 



S140 



Backup the present 
values 



S141 



Get the distance 
between the inserting 
pointer and the reading 
pointer 



IS' 



S142 




C Return 0 ) 



S145 



Calculate the Remaining 
GOV Playback Time by 
skipping the Incomplete 
GOV and the Blank GOV 



S146 



S147 



Set Bitrate Control 
Information structure 



Send Out Bitrate Control 
Information to the 
BitrateAdapter 



S148 



^ { return 1 ) 



17/25 




18/25 




19/25 



FIG. 22 



Bitrate Control 
Decision 



Receive the Bitrate 
Control Information 
from the Link Buffer 



S150 



Compare the current 
Remaining Playback 
Time to the Sliding 
Window 



S151 



S152 



S154 



Equal 

to or larger than the 

Upperbound 
? 



NO 



YES 



S153 



The Decoder's speed 
is slower than the 
current data bitrate 



Equal 
to or lower than the 
Lowerbound 

YES^ 155 



Network bandwidth 
cannot support the 
current data bitrate 



S156 



Network is 
good enough to 
support current 
data bitrate 



Decision: 

Decrease the encoding bitrate 



S157 



S161 



Check the 
polling timer 



Adjust the Sliding Window 



S158 



Notice the Link Buffer about the 
adjustment of Sliding Window 



S159 



Reset the polling timer 



S160 



20/25 



FIG. 23 



Player Data 
Receiving 
Link Buffer 



Player Data 
Receiving 
Link Buffer 



Decoder 
Taking 
Read( ) 


(gov)w 
(gov) 




Network 
Arriving 
Insert( ) 


(gov) 
(gov/ 





> 



28 



Region to 
calculate the Total 
Remaining GOV 
Playback Time 



Decoder 
Taking 
Read( ) 



Network 
Arriving 
Insert( ) 



50 



DEFINITIONS RELATED TO 
LINK BUFFER 




CD 
3= 



CD 

E 



o 
CO 



Q_ 

> 
o 
o 

O) 

cz 
"c 

E 

CD 

err 



Decision Sliding Window 

Total Buffer PlaybackTime 
(100%) 




CALCULATION OF TOTAL REMAINING 
GOV PLAYBACK TIME 



{Upper Decision Threshold 

Upperbound 
Middlevalue 



Upper Step 



Lower Step 



-90% 
-80% 
■70% , 

.Acceptable \ Lowerbound 
-60% Region *fLw er Decision Threshold 

-50% 

-40% 

-30% 

-20% 

-10% 



> Sliding 
Window 



52 



DESIGN DECISION SLIDING WINDOW 



21/25 



-60% 
-50% 
-40% 
-30% 
-20% 
-10% 



=3 



CD 



O 
CD 



O 
OS 



_£5 

> 
o 

CD 

o> 

c: 

*c 
as 
E 

CD 



80% 






70% i 







Initial State 

Total Buffer PlaybackTime 
. (100%) 



Playback start point 



Current Remaining 
GOV Playback Time 



FIG. 24 

(Upper Decision Threshold 



Upperbound ,. 
vv I Sliding 

Middlevalue J Window 
Lowerbound 



] Lower Decision Threshold 



Filling buffer with GOV 




Normal Playback 



Total Buffer PlaybackTime 
(100%) 



Playback start point 



-60% 
-50% 
-40% 
-30% 
-20% 
-10% 




Filling 
buffer 
with GOV 



0 ! 



^fOpper Decision Threshold 

Uppefcound , S|jd . ng 
Window 



l Middlevalue 

"""H Lowerbound 
j Low er Decision Threshold ^ 



CONDITIONS OF 
NORMAL PLAYBACK 



Time 



22/25 



CD 



CD 



O 
CD 



O 
CO 

Q_ 



O 
O 



CO 

E 

CD 

DC 



Network Deterioration 

Total Buffer PlaybackTime 
(100%) 




FIG. 25 

s [Upper Decision Threshold 

Upperb0Und l Sliding 
Middlevalue J Window 

Lowerbound 
v {Tower Decision Threshold 



54 



CONDITIONS OF NETWORK 
DETERIORATION AND 
SLIDING WINDOW MOVEMENT 



New Sliding Window 



CD 
=3 



CD 



O 
CD 



O 
CO 

_Q 
>■» 

_cO 

CL 



o 

(3 



CO 

E 

CD 
DC 



Total Buffer PlaybackTime 
(100%) 



Playback start point 




-50% 
-40% 
-30% 
-20% 
-10% 



Curren 



Remaining 



GOV Playback Time 



Filling 
buffer with 
GOV 




-j Upper Decision Threshold 

Upperbound ] Slidin 9 
HM I Window 

Middlevalue f( New 

J Position) 

Lowerbound 

hreshold 

55 



'[Lower Decision Threshold 



CONDITIONS OF NETWORK 
DETERIORATION AND 
SLIDING WINDOW MOVEMENT 

Time 



23/25 



Decoder's Poor Throughput 
Total Buffer PlaybackTime 
(100%) 




FIG. 26 



^{Upper Decision Threshold 



Upperbound 
Middlevalue 



Sliding 
f Window 



Lowerbound 

Lower Decision Threshold 
^56 



CONDITIONS OF DECODER'S 
POOR THROUGHPUT AND 
SLIDING WINDOW MOVEMENT 



New Sliding Window 
Total Buffer PlaybackTime 
(100%) 




s {Upper Decision Threshold 



Upperbound 
Middlevalue 
Lowerbound 



> Sliding 
Window 



Lower Decision Threshold 



CONDITIONS OF DECODER'S 
POOR THROUGHPUT AND 
SLIDING WINDOW MOVEMENT 



24/25 



FIG. 27 



28 




27 



DataLinkBuffer(Client) | BitrateAdapter(Client) | BitrateAdapter(Server) 



23 



10 



Encoder 



BitrateControlInformation() ■ 



(Reset Polling Timer) 
AdjustSlidingWindowQ \ 



No Need to Change Bitrate) 

(Polling Timer is Triggered?) 

StartPolling() 

PollingQ 

(Receive Polling Packet and\ 
Monitor Streaming Status J 

PollingQ 

End67PoYing(j' 



( Calculate Packet Loss Rate) 

i 

(Make Decision) 



ChangeBitrate() 
ACK() 

^ 1 





(Reset Polling Timer) 



ChangeBitrate() 

L..... A .?. K o 



25/25 



FIG. 28 



Auto-Negotiation on 
Initial Streaming 
Bitrate 



Choose the first 
polling bitrate 



S 



S200 



Start polling 



S 



S201 




YES 



Choose the nearest 
higher bitrate stage 
as the polling bitrate 



S203 



Start another 
polling 



S205 



S206 




S204 



Choose the nearest 
lower bitrate stage 
as the polling bitrate 



S208 



NO 


S207 








set the polling 






bitrate as the 






initial streaming 


-« 




bitrate 






S209 



YES 



